{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cdac199c",
   "metadata": {},
   "source": [
    "# OpenAI强化学习实战（第15期）第3课书面作业\n",
    "学号：114488  \n",
    "**作业内容：**  \n",
    "\n",
    "1.  尝试将以下场景转换成马尔科夫决策过程的四元组，并尝试编写MDP环境代码（可以不写render函数）\n",
    "\n",
    "   考虑下面的迷宫游戏问题，你控制一个机器人在一个二维迷宫中运动，迷宫中有正常的陆地、火坑、石柱、钻石。你可以控制机器人上下左右运动，机器人不能走到迷宫外面，一次最多只能运动一步，如果不小心掉到火坑中，游戏结束，如果找到了钻石，那么可以得到奖励，并且游戏结束！你的目标是通过设计策略，让机器人尽快地找到钻石，获得奖励。\n",
    "\n",
    "   ![maze](https://i.loli.net/2021/11/02/VuM6k4DnPS3XZzp.png)\n",
    "\n",
    "2. 考虑股票交易环境。是否可以构建股票交易的马尔科夫决策过程？如果可以，描述该过程。如果不可以，说说你的理由。\n",
    "\n",
    "## 作业1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e101f4c5",
   "metadata": {},
   "source": [
    "实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "de591fa5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "\n",
    "env = gym.make('MazeWorld-v0')\n",
    "env.reset()\n",
    "env.render()\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff249391",
   "metadata": {},
   "source": [
    "   ![class03](https://i.loli.net/2021/11/02/eBzEkxqRhJtHyQ2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e781c51b",
   "metadata": {},
   "source": [
    "如上图所示：  \n",
    "1. 用红圆圈表示火坑； \n",
    "2. 用灰圆圈表示石柱；\n",
    "3. 用蓝圆圈表示钻石； \n",
    "4. 用金圆圈表示机器人。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81182d4f",
   "metadata": {},
   "source": [
    "实现代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7ee562a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import numpy\n",
    "import random\n",
    "from gym import spaces\n",
    "import gym\n",
    "\n",
    "# logger = logging.getLogger(__name__)\n",
    "\n",
    "class MazeEnv(gym.Env):\n",
    "    metadata = {\n",
    "        'render.modes': ['human', 'rgb_array'],\n",
    "        'video.frames_per_second': 2\n",
    "    }\n",
    "\n",
    "    def __init__(self):\n",
    "        #状态空间,定义15个状态空间，每一个状态对应一个格子，可以看出不是每个格子都是可到达的\n",
    "        self.states = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15] #状态空间\n",
    "        \n",
    "        self.x=[105,235,365,495,105,365,495,105,235,365,495,105,235,365,495]#对应每个状态格子中心位置坐标x\n",
    "        self.y=[320,320,320,320,240,240,240,160,160,160,160, 80, 80, 80, 80]#对应每个状态格子中心位置坐标y\n",
    "        #定义终止状态\n",
    "        self.terminate_states = dict()  #终止状态为字典格式\n",
    "        self.terminate_states[11] = 1\n",
    "        self.terminate_states[12] = 1\n",
    "        self.terminate_states[15] = 1\n",
    "        \n",
    "        #定义动作空间\n",
    "        self.actions = ['n','e','s','w']\n",
    "        #定义奖励\n",
    "        self.rewards = dict();        #回报的数据结构为字典\n",
    "        self.rewards['7_s'] = -1.0\n",
    "        self.rewards['10_e'] = -1.0\n",
    "        self.rewards['8_s'] = -1.0\n",
    "        self.rewards['13_w'] = -1.0\n",
    "        self.rewards['14_e'] = 1.0\n",
    "        #定义状态转移\n",
    "        self.t = dict();             #状态转移的数据格式为字典\n",
    "        self.t['1_s'] = 5\n",
    "        self.t['1_e'] = 2\n",
    "        self.t['2_w'] = 1\n",
    "        self.t['2_e'] = 3\n",
    "        self.t['3_s'] = 6\n",
    "        self.t['3_w'] = 2\n",
    "        self.t['3_e'] = 4\n",
    "        self.t['4_w'] = 3\n",
    "        self.t['4_s'] = 7\n",
    "        self.t['5_n'] = 1\n",
    "        self.t['5_s'] = 8\n",
    "        self.t['6_n'] = 3\n",
    "        self.t['6_s'] = 10\n",
    "        self.t['6_e'] = 7\n",
    "        self.t['7_n'] = 4\n",
    "        self.t['7_s'] = 11\n",
    "        self.t['7_w'] = 6\n",
    "        self.t['8_n'] = 5\n",
    "        self.t['8_s'] = 12\n",
    "        self.t['8_e'] = 9\n",
    "        self.t['9_e'] = 10\n",
    "        self.t['9_s'] = 13\n",
    "        self.t['9_w'] = 8\n",
    "        self.t['10_n'] = 6\n",
    "        self.t['10_s'] = 14\n",
    "        self.t['10_w'] = 9\n",
    "        self.t['10_e'] = 11\n",
    "        self.t['11_n'] = 7\n",
    "        self.t['11_s'] = 15\n",
    "        self.t['11_w'] = 10\n",
    "        self.t['12_e'] = 13\n",
    "        self.t['12_n'] = 8\n",
    "        self.t['13_e'] = 14\n",
    "        self.t['13_w'] = 12\n",
    "        self.t['13_n'] = 9\n",
    "        self.t['14_e'] = 15\n",
    "        self.t['14_w'] = 13\n",
    "        self.t['14_n'] = 10\n",
    "        self.t['15_w'] = 14\n",
    "        self.t['15_n'] = 11\n",
    "\n",
    "        self.gamma = 0.8         #折扣因子\n",
    "        self.viewer = None\n",
    "        self.state = None\n",
    "\n",
    "    def seed(self, seed=None):\n",
    "        self.np_random, seed = seeding.np_random(seed)\n",
    "        return [seed]\n",
    "\n",
    "    def getTerminal(self):\n",
    "        return self.terminate_states\n",
    "\n",
    "    def getGamma(self):\n",
    "        return self.gamma\n",
    "\n",
    "    def getStates(self):\n",
    "        return self.states\n",
    "\n",
    "    def getAction(self):\n",
    "        return self.actions\n",
    "    def getTerminate_states(self):\n",
    "        return self.terminate_states\n",
    "    def setAction(self,s):\n",
    "        self.state=s\n",
    "\n",
    "    def step(self, action):\n",
    "        #系统当前状态\n",
    "        state = self.state\n",
    "        if state in self.terminate_states:\n",
    "            return state, 0, True, {}\n",
    "        key = \"%d_%s\"%(state, action)   #将状态和动作组成字典的键值\n",
    "\n",
    "        #状态转移\n",
    "        if key in self.t:\n",
    "            next_state = self.t[key]\n",
    "        else:\n",
    "            next_state = state\n",
    "        self.state = next_state\n",
    "\n",
    "        is_terminal = False\n",
    "\n",
    "        if next_state in self.terminate_states:\n",
    "            is_terminal = True\n",
    "\n",
    "        if key not in self.rewards:\n",
    "            r = 0.0\n",
    "        else:\n",
    "            r = self.rewards[key]\n",
    "\n",
    "\n",
    "        return next_state, r,is_terminal,{}\n",
    "    def reset(self):\n",
    "        self.state = self.states[int(random.random() * len(self.states))]\n",
    "        while self.state in self.terminate_states:\n",
    "            #print('final state: ',self.state)\n",
    "            self.state = self.states[int(random.random() * len(self.states))]\n",
    "        return self.state\n",
    "    def render(self, mode='human', close=False):\n",
    "        if close:\n",
    "            if self.viewer is not None:\n",
    "                self.viewer.close()\n",
    "                self.viewer = None\n",
    "            return\n",
    "        screen_width = 600\n",
    "        screen_height = 400\n",
    "\n",
    "        if self.viewer is None:\n",
    "            from gym.envs.classic_control import rendering\n",
    "            self.viewer = rendering.Viewer(screen_width, screen_height)\n",
    "            #创建网格世界\n",
    "            self.line1 = rendering.Line((40, 360), (560,360))\n",
    "            self.line2 = rendering.Line((40, 280), (560, 280))\n",
    "            self.line3 = rendering.Line((40, 200), (560, 200))\n",
    "            self.line4 = rendering.Line((40, 120), (560, 120))\n",
    "            self.line5 = rendering.Line((40, 40), (560, 40))\n",
    "            self.line6 = rendering.Line((40, 360), (40, 40))\n",
    "            self.line7 = rendering.Line((170, 360), (170, 40))\n",
    "            self.line8 = rendering.Line((300, 360), (300, 40))\n",
    "            self.line9 = rendering.Line((430, 360), (430, 40))\n",
    "            self.line10 = rendering.Line((560, 360), (560, 40))\n",
    "            #self.line11 = rendering.Line((420, 100), (500, 100))\n",
    "            #创建石柱\n",
    "            self.shizhu1 = rendering.make_circle(30)#30半径\n",
    "            self.circletrans = rendering.Transform(translation=(235,240))#圆心\n",
    "            self.shizhu1.add_attr(self.circletrans)\n",
    "            self.shizhu1.set_color(0.5,0.5,0.5)\n",
    "            #创建第一个火坑\n",
    "            self.kulo1 = rendering.make_circle(30)#40半径\n",
    "            self.circletrans = rendering.Transform(translation=(495,160))#圆心\n",
    "            self.kulo1.add_attr(self.circletrans)\n",
    "            self.kulo1.set_color(1,0,0)\n",
    "            #创建第二个火坑\n",
    "            self.kulo2 = rendering.make_circle(30)\n",
    "            self.circletrans = rendering.Transform(translation=(105, 80))\n",
    "            self.kulo2.add_attr(self.circletrans)\n",
    "            self.kulo2.set_color(1, 0, 0)\n",
    "            #创建钻石\n",
    "            self.gold = rendering.make_circle(30)\n",
    "            self.circletrans = rendering.Transform(translation=(495, 80))\n",
    "            self.gold.add_attr(self.circletrans)\n",
    "            self.gold.set_color(0, 0, 1)\n",
    "            #创建机器人\n",
    "            self.robot= rendering.make_circle(20)\n",
    "            self.robotrans = rendering.Transform()\n",
    "            self.robot.add_attr(self.robotrans)\n",
    "            self.robot.set_color(0.8, 0.6, 0.4)\n",
    "\n",
    "            self.line1.set_color(0, 0, 0)\n",
    "            self.line2.set_color(0, 0, 0)\n",
    "            self.line3.set_color(0, 0, 0)\n",
    "            self.line4.set_color(0, 0, 0)\n",
    "            self.line5.set_color(0, 0, 0)\n",
    "            self.line6.set_color(0, 0, 0)\n",
    "            self.line7.set_color(0, 0, 0)\n",
    "            self.line8.set_color(0, 0, 0)\n",
    "            self.line9.set_color(0, 0, 0)\n",
    "            self.line10.set_color(0, 0, 0)\n",
    "            #self.line11.set_color(0, 0, 0)\n",
    "\n",
    "            self.viewer.add_geom(self.line1)\n",
    "            self.viewer.add_geom(self.line2)\n",
    "            self.viewer.add_geom(self.line3)\n",
    "            self.viewer.add_geom(self.line4)\n",
    "            self.viewer.add_geom(self.line5)\n",
    "            self.viewer.add_geom(self.line6)\n",
    "            self.viewer.add_geom(self.line7)\n",
    "            self.viewer.add_geom(self.line8)\n",
    "            self.viewer.add_geom(self.line9)\n",
    "            self.viewer.add_geom(self.line10)\n",
    "            self.viewer.add_geom(self.shizhu1)\n",
    "            self.viewer.add_geom(self.kulo1)\n",
    "            self.viewer.add_geom(self.kulo2)\n",
    "            self.viewer.add_geom(self.gold)\n",
    "            self.viewer.add_geom(self.robot)\n",
    "\n",
    "        if self.state is None: return None\n",
    "        #self.robotrans.set_translation(self.x[self.state-1],self.y[self.state-1])\n",
    "        self.robotrans.set_translation(self.x[self.state-1], self.y[self.state- 1])\n",
    "\n",
    "        return self.viewer.render(return_rgb_array=mode == 'rgb_array')\n",
    "\n",
    "    def close(self):\n",
    "        if self.viewer is not None:\n",
    "            self.viewer.close()\n",
    "            self.viewer = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75ebc70c",
   "metadata": {},
   "source": [
    "## 作业2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77de3527",
   "metadata": {},
   "source": [
    "考虑股票交易环境。是否可以构建股票交易的马尔科夫决策过程？如果可以，描述该过程。如果不可以，说说你的理由。  \n",
    "\n",
    "我认为股票交易过程可以构建MDP，理由是存在可以构建MDP的要素，下面我以单只股票为例来说明一下：  \n",
    "构成MDP要有四/五元组$(S,A,T,R,\\gamma)$  ，\n",
    "\n",
    "其中$S$表示状态，我认为一种可能的状态模型如下：\n",
    "\n",
    "* B1表示该股票涨幅在0~5%之间。\n",
    "\n",
    "* B2表示该股票涨幅在5~10%之间。\n",
    "* C1表示该股票涨幅在0~-5%之间。\n",
    "\n",
    "* C2表示该股票涨幅在-5~-10%之间。\n",
    "\n",
    "\n",
    "\n",
    "可能的状态如下图所示："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f8aca03",
   "metadata": {},
   "source": [
    "![openai-class03](https://i.loli.net/2021/11/05/rOjP7xvzwFMV829.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34d1d999",
   "metadata": {},
   "source": [
    "$A$表示动作空间，可能的动作空间如下：\n",
    "\n",
    "* 啥也不做，即不买入也不卖出。\n",
    "* 买入一定量股票。\n",
    "* 卖出一定量股票。\n",
    "\n",
    "\n",
    "\n",
    "$T$表示状态转移矩阵，表明了不同状态间的转移概率。\n",
    "\n",
    "\n",
    "\n",
    "$R$表示回报，这个回报可以根据买入或者卖出后投票价格变化给出，对于买入，如果后续股票价格上涨，则为正回报，上涨越多回报值越大，如果下跌，则为负回报，下跌越多负回报越大。\n",
    "\n",
    "\n",
    "\n",
    "$\\gamma$表示折扣率，用来计算回报情况，离交易越远的价值折扣越大。\n",
    "\n",
    "\n",
    "\n",
    "根据上述可以构建MDP过程用来模拟股票交易情况，当然我认为还可以构建更为复杂的MDP过程。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
